The subject matter of this patent application relates to a method of processing a sequence of coded video frames. The method of processing allows the coded frames to be retrieved during playback in an unambiguous manner.
A television programming provider typically produces a programming signal for distribution by a service provider (such as a cable system operator) over a video transmission network to a wide audience of viewers. Conventionally, the programming signal begins as an uncompressed video sequence and at least one corresponding uncompressed audio sequence. The subject matter of this application is concerned with processing the video sequence and accordingly we will not discuss the audio sequence further.
The uncompressed video sequence consists of a series of sequential frames, representing respective images, and is assembled at a production facility. After assembly, the uncompressed video sequence is compressed by a video encoder, which encodes each frame using a video compression algorithm, such as that which is commonly known as MPEG-2, and creates a corresponding compressed video frame. The coded video sequence is transmitted over a transmission network to customer premises at which a video decoder included in a receiving terminal decodes the video sequence for a selected program and supplies the decoded frames to a television set for presenting the corresponding sequence of images to the viewer.
The MPEG-2 video compression algorithm encodes an uncompressed video frame as an intra-coded frame, or I frame, as a predictive-coded frame (P frame) or as a bi-directionally predictive-coded frame (B frame). I frames and P frames are also known as reference frames or anchor frames whereas B frames are also referred to as dependent frames. An I frame contains a complete description of the original picture. A P frame contains a description of the picture compared to a temporally earlier I frame. This allows the encoder to use considerably fewer bits to describe a P frame than would be required for an equivalent I frame. A B frame contains a description of the picture compared to both a temporally earlier reference frame and a temporally later reference frame. This allows the encoder to use approximately an order of magnitude fewer bits to describe a B frame than an equivalent I frame. It will therefore be appreciated that a P frame is both a dependent frame (with respect to an I frame) and a reference frame (with respect to a B frame).
Each coded frame includes, in addition to data representing the captured image, a presentation time stamp. The presentation time stamp, or PTS, is a 33 bit value of the count attained by counter that is counting cycles of a 90 kHz system clock signal. The PTS value reflects the desired playout-time of the frame relative to the system clock.
The sequence of coded frames is input to a system encoder which encapsulates the coded frames in packets (such as packetized elementary stream packets, which are well known to those skilled in the art) that can be efficiently transmitted over suitable communication infrastructure to a receiving terminal that includes a video decoder.
Generally it is intended that the frames should be presented for display in the same order as the corresponding images were acquired, for example by a camera. For each uncompressed video frame, the video encoder determines the appropriate type of the corresponding coded video frame and the coded frame's place in the encoding order. The encoder may determine that a first frame (F1) should be coded as a reference frame (I or P). In this case, the next two frames (F2, F3) will normally be encoded as B frames and the fourth frame (F4) as a P frame. The encoder will first encode frames F1 and F4 and then encode frames F2 and F3 using the encoded frames F1 and F4 as reference frames. The encoder transmits the frames in the sequence F1, F4, F2, F3.
Let us assume that frame F1 is encoded as an I frame. Since a dependent coded frame depends on at least one reference frame, the decoder must decode the reference frame(s) before the dependent frame can be decoded. Therefore, although the coded frames are transmitted, and subsequently decoded, in the encoding order F1, F4, F2, F3, the downstream receiving terminal may not simply output the decoded frames in the order they are received. For coded frames transmitted earlier in the sequence than they are to be displayed, the system encoder inserts a decode time stamp (DTS), relative to the system time clock, into the coded frame's packet in addition to the PTS. For those frames for which no reordering is necessary, the DTS and PTS would be identical and therefore only the PTS is transmitted and the PTS is used to determine the decode time.
The receiving terminal operates in known fashion to generate a system clock signal that is synchronized to the encoder's system clock signal. A receiving terminal with minimal functionality, such as a simple set-top box (STB) without recording capability, comprises a receiver that recovers the sequence of coded video frames from the packetized elementary stream and a video decoder that receives coded the video frames, buffers and decodes the frames based on DTS, and buffers and presents the frames based on PTS.
Many subscribers to cable and satellite television distribution services use a more sophisticated receiving terminal that incorporates a PVR (personal video recorder) to record television program material for later playback and viewing. In this case, the video frames are stored in coded form and are decoded and played back when desired in a similar manner to that employed by the simple STB described above.
A typical PVR supports various trick playback modes, including fast forward (FF) and rapid reverse (RR), which allow a viewer to scan rapidly through material of little interest. The PVR accomplishes FF and RR playback by discarding frames of the received sequence, i.e. by omitting frames of the received sequence from the sequence that is decoded and supplied to the video display buffer. The PVR displays frames at the normal constant rate (i.e. about 30 frames per second in the United States) but since frames of the received sequence are discarded, the displayed image evolves at a greater speed than in normal playback.
The uncompressed video sequence that is compressed by the video encoder may include feature content, such as an episode of a recurring television program, interspersed with supplemental content blocks (e.g. one or more commercials, public service announcements, station identification messages, etc.). At the production facility, the programming provider uses conventional video editing techniques to insert the supplemental content blocks into the feature content at predetermined intervals.
The supplemental content blocks that are inserted into the uncompressed video sequence at the production facility typically take the form of a series of video sequences having relatively short duration (e.g. 8 distinct video sequences each having a duration of 30 seconds or 1 minute). As part of a commercial arrangement between the programming provider and the service providers, some advertising content blocks may contain some low priority advertising content, such as advertisements provided by the national television network itself. This allows the regional or local service providers to overwrite the low priority advertising content in the programming signal with their own local or more specifically targeted advertising content in the form of sequences of coded video frames. This ‘ad-insertion’ capability is advantageous for the service providers because they can provide targeted advertising content specifically aimed at their customer base.
However, insertion of advertising content blocks in the sequence of coded video frames may result in discontinuities in timestamps (PTS and DTS).
When the coded video sequence content is decoded and presented as received, as by the simple STB described above, the discontinuities in timestamps are hidden. However, when the coded video sequence material is recorded by a PVR and played back later, and the user wishes to use trick play capabilities, several undesirable effects may be observed. In particular, it is difficult for the PVR to select the correct video frames and send them to the decoder when there are discontinuities in the time stamps. Further, pausing and subsequently resuming at the previously paused frame is difficult to achieve, and so is slow forwarding frame by frame, in the case of content with discontinuous time stamps.
More sophisticated receiving terminals (with PVR capability) are subject to limited control by the service provider (such as a cable system operator), allowing the service provider to store supplemental content for later insertion into a video stream being played back and displayed. For example, the service operator may cause the receiving terminal to ingest advertising material from the internet for subsequent play out. In most cases, such material is ingested at non-real time speeds.
It has been proposed that a PVR should store the coded video frames in the mass storage device using a relational database having a content file and an index file. In an implementation of this proposal, the PVR receives the video frames and assigns a local frame time stamp (LFTS) to each frame, stores the video frames in the content file along with content offset position (the record number of the frame in the content file) to identify the location of the frame in the content file, and stores the LFTS in the index file along with the PTS and offset information to identify the location. On playback, the DVR uses the PTS to retrieve an LFTS from the index file and then uses the retrieved LFTS to determine the temporal position and offset of the relevant video frame from the content file (so as to recreate the sequence of frames as received), buffers and decodes the retrieved frames based on DTS, and buffers and presents the decoded frames based on PTS.
One problem with this known DVR is that when an ad is inserted, the frames of the ad have their own sequence of PTS values so there is an interruption in the sequence that started at the beginning of the program. Another is that during trick play the PTS values are not continuously increasing.
Use of a local frame time stamp in this manner is not optimum because PTS discontinuities may result from reusing of PTS values by multiple programs and interleaving of different video streams.
Moreover, this approach is not helpful when recording happens at non real-time speeds, as discussed above, because the LFTS value is based on an operating system clock tick in the set-top box and not on the real time of the video stream. Thus, the temporal position based on LFTS value may not be accurate.